Many scalable video-coding approaches have been proposed over the past few years for real-time Internet applications. In addition, several video scalability approaches have been adopted by video compression standards such as MPEG-2, MPEG-4, and H.26x. Temporal, spatial, and quality (SNR) scalability types have been defined in these standards.
All of these types of scalable video consist of a so-called base layer and one or multiple enhancement layers. The base layer part of the scalable video stream represents, in general, the minimum amount of data needed for decoding that stream. The enhancement layer part of the stream represents additional information, and therefore it enhances the video signal representation when decoded by a receiver. For each type of video scalability, a certain scalability structure is used. The scalability structure defines the relationship among the pictures of the base-layer and the pictures of the enhancement layer.
Another type of scalability, which has been primarily used for coding still images, is fine granular scalability (FGS). Images coded with this type of scalability can be decoded progressively. In other words, the decoder can start decoding and displaying the image after receiving a very small amount of data. As more data is received, the quality of the decoded image is progressively enhanced until the complete information is received, decoded, and displayed.
The FGS encoding framework provides a good balance between coding-efficiency and a very simple scalability structure. As shown in FIG. 1, the FGS structure consists of two layers: a base-layer coded at a bitrate RBL and an enhancement-layer coded using a fine-granular (or embedded) scheme to a maximum bitrate of Rmax. FIG. 1 shows examples of the FGS scalability structure at the encoder (left), streaming server (center), and decoder (right) for a typical unicast Internet streaming application. The top and bottom rows of the figure represent base-layers without and with Bi-directional (B) frames, respectively.
This structure provides a very efficient, yet simple, level of abstraction between the encoding and streaming processes. The encoder only needs to encode the video as a base layer and an enhancement layer, and it does not need to be aware of the particular bitrate at which the content will be streamed. The streaming server, on the other hand, has a total flexibility in sending any desired portion of any enhancement layer frame (in parallel with the corresponding base layer picture), without the need for performing complicated real-time transcoding algorithms. This enables the server to handle a very large number of unicast streaming sessions, and to adapt to their bandwidth variations in real-time. On the receiver side, the FGS framework adds a small amount of complexity and memory requirements to any standard motion-compensation based video decoder. These advantages of the FGS framework are achieved while maintaining rather surprisingly good coding-efficiency results.
One of the problems that continue to present itself in the context of streaming application is that of limited and/or fluctuating bandwidth. That is, as congested networks such as the Internet continue to find wide and varied use, bandwidth can become limited and can fluctuate during periods of higher and lower usage. The sending rate of video stream has to be adjusted accordingly. As a result, “jittering” can be very annoying to most video viewers. Accordingly, streaming vendors such as video vendors endeavor to provide constant or smooth quality. Due to the burstiness of video streams and bandwidth fluctuations of the transmission media, achieving this goal can be very challenging.
FGS coding provides the possibility of adapting a video stream taking into account the available bandwidth. However, FGS coding scheme itself does not provide for any smoothing techniques when bandwidth decreases sharply. This becomes especially important when consideration is given to the enhancement layers that contain important content in FGS steaming applications. One straightforward utilization of FGS coding in the context of limited bandwidth situations can involve simply dropping portions of the enhancement layer when bandwidth becomes limited in order to decrease the bit rate of the data stream. But, when bandwidth sharply decreases, as is often the case, too much of the enhancement layer can be dropped and there is no way to guarantee the quality of the video. That is, as bandwidth decreases when important content is being streamed, there is no protection scheme to prevent the important content from being dropped.
Accordingly, this invention arose out of concerns associated with providing improved streaming methods and systems. In particular, this invention arose out of concerns associated with providing methods and systems for scalable streaming.